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Abstract 

A  new  paradigm  for  automatic  non-photorealistic  render¬ 
ing  is  introduced  in  this  paper  Non-photorealistic  render¬ 
ing  (NPR)  provides  an  alternative  way  to  render  complex 
scenes  by  emphasizing  high  level  or  salient  perceptual  fea¬ 
tures.  Particularly,  the  pen-and-ink  rendering  style  pro¬ 
duces  sketchy-like  drawings  that  can  effectively  communi¬ 
cate  shape  and  geometry.  This  is  achieved  by  combining 
drawing  primitives  that  mimic  ink  patterns  used  by  artists. 
Existing  NPR  approaches  can  be  categorized  in  two  groups 
depending  on  the  type  of  input  they  use:  image  based  and 
object  based.  Image  based  NPR  techniques  use  2D  images 
to  produce  the  renderings.  Object  based  techniques  work 
directly  on  given  3D  models  and  make  use  of  the  full  vol¬ 
umetric  representation.  In  this  paper  we  propose  to  enjoy 
the  best  of  both  worlds  developing  an  hybrid  model  that  si¬ 
multaneously  uses  information  from  the  image  and  object 
domains.  These  two  sources  of  information  are  provided 
by  a  calibrated  stereoscopic  system.  Given  a  pair  of  stereo 
images  and  the  calibration  data  we  solve  the  stereo  prob¬ 
lem  in  order  to  extract  the  normal  and  principal  direction 
fields,  which  are  fundamental  to  guide  a  texture  synthesis 
algorithm  that  generates  the  NPR  renderings.  In  particular, 
normals  guide  tonal  variations,  while  principal  directions 
determine  the  orientation  of  stroke-like  texture  patterns.  We 
describe  a  particular,  fully  automatic,  implementation  of 
these  ideas  and  present  a  number  of  examples. 

1.  Introduction 

Non-photorealistic  rendering  can  produce  schematic  repre¬ 
sentations  of  scenes  that  include  only  the  relevant  geomet¬ 
ric  information,  clarifying  shapes  and  focusing  attention. 
This  means  simple  sketches  that  suffice  to  visually  com¬ 
municate  geometry  and  omit  secondary  scene  information. 
Mimicking  artists  abilities  involved  in  the  drawing  process 
well  defined  formation  rules  can  be  deducted  and  then  re¬ 
produced  by  automatic  computer  graphics  algorithms.  For 
example,  pen-and-ink  drawings  can  be  artificially  generated 
by  adequately  placing  individual  strokes  over  the  image,  see 


[21,  18,  22]  and  references  therein.  Varying  the  density  and 
direction  of  strokes  a  wide  range  of  textures  can  be  achieved 
that  represent  different  materials,  illumination  conditions, 
shapes,  etc. 

Existing  NPR  approaches  can  be  categorized  in  two 
groups  depending  on  the  type  of  input  they  use:  image  or 
object  based.1  Image  based  approaches  use  2D  gray  level 
information  to  generate  the  renderings,  e.g.  [18, 19].  As  any 
type  of  image  can  be  used  much  more  complex  models  can 
be  considered  (e.g.  real  scenes,  landscapes,  faces,  etc.)  but 
no  geometric  information  can  be  inferred  from  the  images 
themselves.  On  the  other  hand,  object  based  techniques 
work  directly  with  3D  models,  e.g.  [22,  6,  16,  13,  10,  17]. 
They  have  full  access  to  the  geometry  (which  is  essential 
to  represent  shape)  but  are  limited  to  computer  generated 
scenes. 

We  then  propose  to  have  the  best  of  both  worlds,  consid¬ 
ering  an  hybrid  system  where  images  and  the  geometry  are 
simultaneously  available.  These  two  sources  are  obtained 
from  a  calibrated  stereoscopic  system  that  provides  the  im¬ 
ages  and  the  corresponding  3D  reconstruction  as  inputs  to 
guide  the  rendering.  Although  stereo  reconstruction  may 
not  be  accurate  enough  for  many  applications,  we  show  that 
the  shape  information  needed  to  generate  pen-and-ink  illus¬ 
trations  can  be  obtained  from  a  stereoscopic  system. 

Regarding  the  rendering  stage,  there  are  two  basic  illus¬ 
tration  principles  that  properly  combined  convey  the  desired 
appearance  to  pen-and-ink  drawings:  density  and  orienta¬ 
tion  of  strokes.2  Varying  the  hatch  density  we  can  represent 
different  tonalities  or  illumination  conditions  ranging  from 
bright  (no  hatching),  to  dark  or  shadowed  areas  (densely 
crosshatched).  Orientation  carries  the  shape  information, 
and  stroke  directions  that  follow  the  projection  of  princi¬ 
pal  directions  of  the  surface  are  known  to  be  appropriate 


Another  possible  categorization  of  NPR  algorithms  is  according  to  the 
degree  of  user  intervention  they  require.  Some  just  assist  the  user  in  the 
generation  process,  others  (the  vast  minority)  are  fully  automatic.  The 
one  we  propose  here  belongs  to  the  small  yet  very  important  class  of  fully 
automatic  algorithms. 

2  Additional  features  that  further  enhance  the  drawings  may  include  sil¬ 
houette  drawing,  edge  tracing,  etc. 
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in  representing  shape  [14].  This  is  the  approach  taken  in 
[17,  10,  13,  16]  and  the  one  we  follow  here.  Given  then  a 
target  tonal  value  and  a  preferred  orientation,  what  the  ren¬ 
dering  process  does  is  find  the  appropriate  combination  of 
strokes  that  achieves  that  particular  configuration.  The  tar¬ 
get  tone  and  orientation  combinations  are  obtained  from  the 
stereoscopic  system. 

1.1.  Algorithm  Overview 

The  main  steps  of  the  proposed  NPR  from  stereo  algorithm 
are  the  following: 

1.  Reconstruct  the  3D  surface  given  the  pair  of  images 
and  the  calibration  data.  Once  we  have  the  surface, 
normals  and  principal  directions  can  be  readily  ob¬ 
tained.  This  is  addressed  in  Section  2. 3 


Each  point  mi  =  (u,  v)  in  the  first  image  has  its  correspon¬ 
dent  m2  =  =  (u  +  U(u,  v),  v  +  V(u,v))  in  the 

second  image,  where  U(u,v )  and  V(u,v)  are  the  so  called 
horizontal  and  vertical  disparities.  If  disparities  are  known, 
3D  coordinates  can  be  analytically  obtained  by  solving  the 
stereo  reconstruction  formulas.  We  compute  disparities  us¬ 
ing  a  variational  energy  approach  that  recovers  a  dense  dis¬ 
parity  map  from  a  set  of  two  weakly  calibrated  stereoscopic 
images  [1].  We  can  then  solve  the  reconstruction  formu¬ 
las  and  get  the  surface  coordinates  in  parametrized  form: 
S(u,v)  =  (x(u,v),y(u,v),z(u,v)).  Taking  directional 
derivatives  Su  =  (xu,yu,zu)  and  Sv  =  (xv,yv,zv),  3D 
normals  are  obtained  from: 


N  (u,  v ) 


SU  A  Sy 
|<5u  A  Sy\ 


(1) 


2.  Compute  required  tone  and  orientation  combinations 
from  the  geometry  and  image  domains  provided  by  the 
stereoscopic  system.  These,  which  are  just  particular 
examples  of  the  information  that  can  be  extracted  from 
the  stereo  data,  are  further  described  in  Section  3. 

3.  The  last  step  in  the  pipeline  is  the  rendering  itself.  Here 
we  use  a  texture  synthesis  algorithm  guided  by  the  il¬ 
lumination  and  orientation  values  computed  in  the  pre¬ 
vious  step.  This  is  described  in  Section  4. 

Given  a  calibrated  image  pair,  the  reconstructed  surface 
and  its  differential  properties  (normals  and  principal  direc¬ 
tions)  can  be  obtained  in  many  ways.  Likewise,  the  ren¬ 
dering  stage  can  be  performed  using  various  existing  tech¬ 
niques  ( e.g .  individual  stroke  placement,  texture  synthe¬ 
sis,  etc).  What  is  important  and  the  main  idea  we  want  to 
present  here  is  the  concept  of  combining  both  sources  of 
information  (2D  and  3D)  as  input  to  guide  the  synthesis  of 
non-photorealistic  renderings.  As  an  example,  we  describe 
in  this  paper  a  particular  realization  of  a  system  of  this  kind. 
Various  examples  are  presented  in  Section  5. 

2.  Solving  the  Stereo  Problem 

Given  a  stereo  pair  of  images  (Iuh)  and  corresponding 
calibration  data  we  can  reconstruct  3D  scene  coordinates 
up  to  a  given  transformation.  The  type  of  reconstruction 
(Euclidean,  affine,  or  projective)  depends  on  the  calibration 
data  available.  We  will  assume  that  intrinsic  and  extrin¬ 
sic  camera  parameters  are  known  so  Euclidean  reconstruc¬ 
tion  can  be  performed.  See  [8,  9,  11]  for  details.  Any  3D 
space  point  X  =  (x,y,z)  projects  into  both  camera  retinal 
planes  as  mi(u,v)  E  I\  and  rri2(u\v')  E  I2 ,  where  (u,v) 
and  are  2D  coordinates  expressed  in  the  systems  at¬ 

tached  to  the  first  and  second  image  planes,  respectively. 

3 Note  that  in  certain  scenarios,  normals  and  principal  directions  can  be 
obtained  without  a  full  3D  reconstruction. 


Principal  curvatures  and  directions  are  the  eigenvalues  and 
eigenvectors  of  the  shape  operator  I-1  II,  where  I  and  II 
are  the  first  and  second  fundamental  forms  of  the  surface: 


I  = 


11  = 


Su  •  Su 
SU  ’  Sy 

Suu'N 

Suv  •  N 


SU  •  Sy 
Sy  *  Sy 


Suv'  N 
Syy  •  N 


(2) 


The  one  we  just  described,  is  the  straight  forward  ap¬ 
proach  to  compute  normals  and  principal  directions  from 
stereo.  Another  approach  may  be  to  obtain  normals  and 
principal  directions  directly  from  the  images  without  actu¬ 
ally  reconstructing  the  3D  surface.  This  was  done  in  [3],  but 
poor  principal  curvature  estimates  were  obtained.  A  third 
alternative  is  to  follow  the  variational  approach  in  [7]  that 
directly  computes  the  surface  (in  implicit  form)  given  an 
arbitrary  number  of  images.  Having  an  implicit  represen¬ 
tation  of  the  surface  makes  it  easier  to  obtain  normals  and 
principal  directions. 


3.  Guiding  the  Rendering  Stage 

This  section  describes  how  to  obtain  tone  and  orientation 
values  to  guide  the  rendering  stage  using  image  and  object 
domain  features  provided  by  the  stereoscopic  system.  Be¬ 
fore  starting,  we  note  that  the  reconstructed  surface  S(u,  v ) 
is  somewhat  noisy,  so  a  smoothing  step  is  required  before 
computing  any  of  the  differential  quantities  presented  in  the 
previous  section.  As  pointed  out  in  [13],  important  scene 
features  are  better  captured  by  tone  variations,  while  direc¬ 
tions  only  provide  a  low  resolution  field  that  indicates  gen¬ 
eral  object  shape.  Accordingly,  we  perform  a  small  amount 
of  smoothing  before  computing  normals  (so  we  keep  rel¬ 
evant  features),  and  apply  a  bigger  amount  of  smoothing 
before  computing  principal  directions  (so  we  obtain  a  lower 
resolution  direction  field). 
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3.1.  Computing  Illuminations 

Tone  values  at  each  image  point  are  obtained  by  project¬ 
ing  surface  normals  into  the  viewing  direction.  This  gives 
bright  values  for  fronto-parallel  surface  points  and  increas¬ 
ingly  dark  values  for  points  approaching  object  silhouettes. 
We  can  also  exploit  the  image  based  side  of  our  approach 
and  extract  illumination  features  from  the  images  them¬ 
selves.  Although  we  could  use  any  of  the  techniques  re¬ 
ported  in  image  based  rendering,  for  the  examples  in  this 
paper  we  simply  detect  dark  image  regions  by  threshold¬ 
ing.4  Other  relevant  image  features  may  include  texture, 
edges,  object  segmentation,  etc.5  Feature  extraction  can  be 
made  more  robust  by  combing  estimates  from  both  stereo 
images,  we  have  not  explored  this  option,  but  it  can  cer¬ 
tainly  improve  the  results.  By  merging  illumination  infor¬ 
mation  extracted  from  both  sources,  we  obtained  the  target 
tone  image  that  will  guide  the  rendering.  We  now  describe 
how  to  get  the  target  orientations. 

3.2.  Computing  Orientations 

As  mentioned  in  Section  1  directions  that  follow  the  projec¬ 
tion  of  principal  directions  of  the  surface  are  known  to  be 
appropriate  in  representing  shape.  We  already  know  how 
to  compute  principal  directions  from  stereo,  except  that  at 
umbilic  points  they  are  not  defined  (all  directions  are  prin¬ 
cipal),  so  solving  the  eigenvalue  problem  is  ill-posed.  In 
the  actual  computation  we  get  a  condition  number  that  indi¬ 
cates  presence  or  proximity  to  an  umbilic  point.  We  there¬ 
fore  discard  the  unreliable  estimates  and  provide  a  method 
for  filling-in  the  missing  values.  A  similar  approach  was 
taken  in  [13].  In  the  end  we  need  to  project  3D  principal  di¬ 
rections  into  the  image  plane,  therefore,  we  choose  to  apply 
the  fill-in  procedure  after  the  projection  step  and  the  par¬ 
ticular  technique  we  use  is  based  on  the  variational  vector 
diffusion  approach  [20] . 

Let  U (u,v)  be  the  (projected)  principal  direction  field 
estimate  obtained  by  solving  the  eigenvalue  problem  de¬ 
scribed  in  Section  2.  A  regularized  U  field  is  then  obtained 
as  the  minimizer  of  the  following  energy: 

£7(U)  =  1  f  w(U-U)2  +  (l-w)||VU||2dudv  (3) 
2  Jn 

The  first  component  in  this  equation  is  a  fidelity  term  that 
forces  the  minimizer  to  be  close  to  the  estimate  U.  The 
second  component  is  the  regularizing  term  that  guarantees 
smoothness.  The  weighting  factor  uj  accounts  for  the  accu¬ 
racy  of  the  estimate  U,  the  idea  is  to  leave  accurate  values 

4 Gray  level  encoded  features  are  clearly  overseen  when  reconstructing 
geometry,  but  are  easily  recognized  in  the  image  domain. 

5  Through  out  this  work  we  assumed  a  good  segmentation  of  the  ob¬ 
ject  was  given,  meaning  we  can  separate  the  object  from  the  background. 

Instead,  we  could  have  obtained  the  segmentation  automatically  from  the 
images. 


unchanged  (u  =  1)  while  allowing  those  that  are  unreli¬ 
able  to  evolve  (cj=0).  Computing  the  Euler-Lagrange  for 
the  energy  in  Equation  (3)  we  obtain  the  minimizing,  unity¬ 
preserving,  evolution: 

U t=  cj(U  —  (U  •  U)U)+ 

(1  -  cj)(AU  -  (AU  •  U)U)  (4) 
U(0)  =  u 

The  steady  state  solution  of  this  system  will  provide  a  suit¬ 
able  2D  direction  field  for  guiding  the  rendering  stage. 

At  this  point  we  have  completely  determined  target  tone 
and  orientations  by  simultaneously  using  image  and  object 
space  features.  When  actually  doing  the  synthesis  we  will 
only  have  a  discrete  number  of  different  tones  (dictated  by 
the  particular  texture  set  we  are  using),  so  we  need  to  thresh¬ 
old  illumination  values  into  the  available  number  of  tones.6 
The  same  happens  for  orientation  values,  we  only  have  a 
discrete  number  of  possible  orientations.  For  the  examples 
in  Section  5  we  use  6  and  8  different  illumination  levels 
(depending  on  the  texture  set)  and  180  orientation  values. 

4.  Non-photorealistic  Rendering 

Two  methods  can  be  used  to  generate  stroke  based  image 
renderings.  The  first  is  to  individually  place  each  stroke 
on  the  target  image  until  the  desired  target  appearance  is 
reached.  Bright  areas  of  the  image  would  require  fewer 
strokes,  less  density,  while  darker  or  shadow  areas  would 
require  denser  stroke  concentrations.  To  achieve  higher 
density  levels  we  can  decrease  the  distance  among  adjacent 
strokes  or  we  can  even  superpose  them  in  different  direc¬ 
tions  (cross  hatching).  Systems  that  used  individual  placing 
of  stokes  were  studied  in  [18,  21,  22,  13]. 

The  second  approach  is  to  use  a  set  of  pre-generated 
stroke  textures  and  apply  a  texture  synthesis  algorithm  to 
reproduce  those  patterns  throughout  the  image.  By  using 
different  texture  sets  a  variety  of  rendering  styles  can  be 
obtained.  This  is  the  approach  used  in  [16,  10]  for  object- 
space  rendering  (texture  mapping  on  3D  surfaces).  This 
simpler  approach  is  easy  to  implement  and  is  the  one  we 
use  here.  Observe  that  we  only  need  to  select  a  basic  set  of 
textures  and  then  run  a  standard  texture  synthesis  algorithm, 
we  describe  the  specific  algorithm  in  the  next  section. 

For  every  possible  combination  of  tone  and  orientation 
we  need  to  provide  a  sample  texture  that  achieves  that 
particular  configuration.  Textures  with  different  tones  are 
achieved  by  varying  the  density  of  strokes,  we  then  simply 
pre-rotate  each  of  them  to  get  all  the  possible  orientations. 
This  way  we  have  a  2D  bank  of  texture  samples,  each  rep¬ 
resenting  a  particular  combination  of  tone  and  orientation, 
see  Figure  1 .  Given  a  particular  target  tone  and  orientation 

6 Thresholds  are  computed  by  minimum  variance  quantization. 
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Figure  1:  Image  bank  generated  with  texture  sets  taken  from  [16].  Di¬ 
rections  span  along  the  horizontal  axis  and  illumination  values  along  the 
vertical  axis. 


we  have  to  choose  the  corresponding  texture  sample  from 
the  bank  to  perform  the  synthesis. 

4.1.  Texture  Synthesis  Rendering 

Among  many  texture  synthesis  algorithms  considered  [2,  5, 
4,  12,  15],  patch  based  algorithms  probed  to  be  the  most 
effective  for  the  type  of  textures  we  are  interested  in.  Patch- 
based  texture  synthesis  algorithms  generate  the  output  im¬ 
age  by  pasting  together  patches  of  the  sample  texture.  Two 
such  algorithms  are  [4]  and  [15],  we  mainly  follow  the  lat¬ 
ter. 

Let  Id,i(u ,  v)  denote  the  sample  texture  corresponding  to 
direction  value  d  and  illumination  value  i  (both  take  integer 
values).7  Isynth{u,v )  will  denote  the  output  synthesized 
image.  Pk  is  a  square  patch  of  Idfau,v)  of  size  w  x  w. 
For  each  patch  Pk  we  denote  its  boundary  zone  by  Bpk. 
Ek  denotes  a  square  patch  of  Isynth{u,v)  of  size  w  x  w 
with  boundary  zone  Bpk .  The  boundary  zones  represent  the 
overlapping  regions  between  adjacent  patches,  see  Figure  2. 

The  patch  P \  matches  if  the  distance  between  their 
overlapping  boundary  zones  is  below  some  threshold  value 
e.  The  set  of  matches  for  the  patch  E *.  is  then  defined  as: 

fa  =  {Pk  £  Id,i(u,v)/d(Bpk,  BEk)  <  e}  (5) 

With  these  definitions,  the  patch  based  algorithm  works 
as  follows: 


7 Note  that  unlike  classical  texture  synthesis  here  we  require  a  set  of 
sample  textures  instead  of  just  one  reference  texture. 


Figure  2:  Patch  based  algorithm.  Left:  three  patches  were  already  as¬ 
signed  to  the  output  image  Isynth  iui  v)>  we  now  l°ok  for  the  best  match¬ 
ing  patch  to  paste  in  the  lower  right  corner.  Right:  candidate  patches  are 
searched  in  the  sample  image  Id,i(u,  v)  by  matching  the  overlapping  re¬ 
gions  (shaded  areas). 


•  Randomly  select  a  patch  from  the  corresponding  sam¬ 
ple  image  Id,i  (u,  v )  and  paste  it  in  the  upper  left  corner 

of  Isynth  (^5  P)  • 

•  Build  the  set  fa  of  all  patches  Pk  E  Idfau, v )  whose 
boundary  zone  Bpk  matches  the  current  position  patch 
boundary  Bpk .  Select  one  patch  from  fa  at  random 
and  paste  it  into  the  output  image  Isynth{u ,  v). 

•  Repeat  until  the  whole  image  Isynth{u ,  v)  is  covered. 

To  build  the  set  (pk  the  search  is  done  using  an  approxi¬ 
mate  p-nearest  neighbor  algorithm  for  which  fast  structures 
and  routines  are  readily  available.  The  best  candidate  is 
chosen  at  random  from  these  p-nearest  neighbors  and  pasted 
into  the  output  Isynth(ui  v).  The  search  space  dimension  is 
the  number  of  pixels  in  Bpk  and  the  whole  sample  image 
space  is  considered. 

5.  Data  Sets  and  Examples 

We  used  three  calibrated  stereo  data  sets.  The  first  two  are 
real  faces  and  the  third  one  is  a  concrete  bust  of  Buffalo 
Bill.  Stereo  pairs  and  corresponding  renderings  are  shown 
in  Figures  4,  5  and  6. 

The  synthesis  stage  was  run  with  two  different  texture 
sets.  Set  1  was  obtained  from  [16]  (see  Figure  1),  the  sec¬ 
ond  set  is  shown  in  Figure  3.  These  sets  have  6  and  8  dif¬ 
ferent  tone  values  respectively,  and  were  pre-rotated  to  1 80 
different  directions. 

6.  Summary  and  Discussion 

We  presented  an  automatic  system  for  the  generation  of 
pen-and-ink  illustrations  given  a  calibrated  stereoscopic 
system.  We  simultaneously  use  gray  level  information  from 
the  images  as  well  as  geometric  features  extracted  from  the 
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Figure  3:  Texture  set  2. 


stereo  reconstruction  to  guide  a  texture  synthesis  algorithm 
that  generates  the  renderings. 

The  main  idea  is  general  in  the  sense  that  we  could  have 
used  any  stereo  reconstruction  procedure  and  any  rendering 
technique  (e.g.  individual  placement  of  strokes,  other  tex¬ 
tures  sets  for  the  synthesis,  etc.)  to  obtain  the  results.  The 
emphasis  is  in  the  idea  of  using  both  sources  of  information 
to  guide  the  rendering  stage  and  the  fact  that  a  stereoscopic 
system  can  provide  them  both  with  enough  accuracy  for  this 
particular  task.  Although  the  required  minimum  to  perform 
the  3D  stereo  reconstruction  are  two  images,  if  more  are 
available,  better  3D  models  can  be  obtained,  therefore  bet¬ 
ter  geometry  estimates  would  guide  the  rendering  stage. 

Let’s  conclude  by  mentioning  that  the  approach  in  [19] 
is  similar  in  nature  to  ours,  the  difference  being  that  a  single 
image  is  used.  Therefore,  external  user  action  was  needed  to 
provide  the  directional  fields  that  guide  the  rendering.  We 
propose  to  get  those  fields  from  the  stereo  reconstruction 
without  requiring  any  user  input,  thus  presenting  a  fully  au¬ 
tomatic  approach. 
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Figure  4:  First  face  data  set.  The  stereo  pair,  render¬ 

ings  using  the  first  and  second  texture  sets  and  two  corre¬ 
sponding  close-ups  views  are  shown.  This  data  set  is  dis¬ 
tributed  with  the  Stereo  Flow  software  package  available  at 
http  :  //serdis  .  dis  .ulpgc  .  es/~j  sanchez/research/sof  tware/StereoFl 
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Figure  6:  Buffalo  Bill  bust.  The  stereo  pair,  renderings  using  the 
first  and  second  texture  sets  and  two  corresponding  close-ups  views 
are  shown.  This  data  set  is  available  from  the  GRASP  Laboratory  at 
http  :  / /www .  cis  .upenn .  edu/ ~  j  anem/ rsrch/ dataset /ReadMe  .  html, 


Figure  5 :  Second  face  data  set.  The  stereo  pair,  renderings  using  the  first 
and  second  texture  sets  and  two  corresponding  close-ups  views  are  shown. 
This  data  set  is  available  from  the  RobotVis  project  at  the  INRIA  website. 
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